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[57] ABSTRACT 

A method and system are providedfjor automatically 

locating an object at a vision station by performing an 



edge-detecting algorithm on at least a portion of the 
-gray-scaje^igifeed imageiof th^bjSct. Preferably, the 
algorithm comprises an implementation of the Hough 
transform which includes the iterative application of a 
direction-sensitive, edge-detecting convolution to the 
digital image. Each convolutioiris"a^lie^Avith a differ- 
ent/convolution -maslToTfilTer, each of which is calcu- 
lated to give maximumropon^toluTttige of -the objecJ 
^ a~_di ffcTent diTOtioirThe method and system have 
the abilitytoTextract edges from low contrast images. 
Also, preferably, a systolic array processor applies the 
convolutions. The implementation of the Hough trans- 
form also includes the steps of shifting the resulting 
edge-enhanced images by certain amounts-hvthe hori- 
zontal aj^jS^rticid-bUrecti^ shifted im- 
'agesltogether into an accumulator buffer to obtain an 
accumulator image and detecting the maximum re- 
sponse in the^accumulator image which correspondssto 
Ltfellocation^of an edge. If the object to be found is 
permitted to rotate, at least one other feature^such as 
another edge, must be located in ordeoorspecify the 
l^atiojrand orientation of the object. The location of 
the object when correlated with the nominal position of 
the object at the vision statiojrproyide s the - positforrana? 
attimd^^e objec^The resultant data may be subse- 
quentiT^transformed into, the coordinate frame of a 
peripheral device, such as a robot, programmable con- 
troller, numerical controlled machine, etc. for subse- 
quent use by a controller of the peripheral device. 

13 Qaims, 11 Drawing Figures 
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F/g.5 



VISION PROGRAM (RUN-TIME MODULE) 



( START ) 




TRAINING SEQUENCE 



C START J 



PLACE SAMPLE OBJECT IN VIEW OF CAMERA 



TRAIN EACH FEAT URE TO BE FOUND 

V ■ 

PLACE OBJECT IN NOMINAL POSITION, FIND 
IT WITH THE VISION SYSTEM, AND STORE 
THE FOUND LOCATIONS AS THE NOMINAL 
POSITION. ^ 

C END ) 



Fig. 6 
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SETUP R-TABLE (TEMPLATE FOR HOUGH TRANSFORM) 



( START ) 







CHOOSE REFERENCE POINT 






CHOOSE FILTER SIZE AND a FOR GAUSSIAN 






CHOOSE EDGE POINTS 







SELECT FIRST EDGE POINT 



COMPUTE AND NORMALIZE CONVOLUTION 
FILTER FOR EDGES IN THIS DIRECTION 
AND STORE IN R-TABLE 




)(ENT DIRECTION FOR 
, STORE IN R-TABLE 



SELECT 
NEXT EDGE 
POINT 




NO 



( END ) 



Fig. 7 
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Fig. 8 



SET UP CONVOLUTION FILTER 



( START ) 



FOR EACH , i* f..n, J»l..n t n= FILTER 

DIMENSION, COMPUTE v - I - ^ , h=j- ^ 
aji/; : -v cos9-h si'n8 P o 



2cr2 



MULT/PLY ALL FILTER 
ELEMENTS BY K,K>I 



PERFORM CONVOLUTION ON 
WINDOW OF IMAGE ANO TAKE 
HIS TOGRAM OF RESULT 



YES 



MULTIPLY ALL FILTER 
ELEMENTS BY K,K<f 



YES 



HIGHEST 
RESULT PIXEL 
< MAXIMUM ALLOWED 
GREY LEVEL 
(255) 



HIGHEST 
RESULT PIXEL- _ 
MAX GREY LEVEL(255£ 
AND > I PIXEL AT ' 
THIS LEVEL. 



NO 



DONE. FILTER GAIN FACTOR HAS 
BEEN SET AUTOMATICALLY FOR 
IMAGE WITHIN WINDOW 



c ) 
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CALIBRATION 
( STl 


PROCEDURES 
\RT ) 


MOVE ROBOT TO NOMINAL POSIT/ON I 






ALIGN OBJECT WITH ROBOT GRIPPER 1 






PLACE CURSORS ON TWO FEATURES OF THE OBJECT 






MOVE ROBOT INTERACTIVE^ FROM THE VISION 
SYSTEM WITHOUT ROTATION 







PLACE CURSORS ON SA ME TWO FEATURES OF THE OBJECT | 

* • 

MOVE ROBOT INTERACTIVELY AGAIN , TO PO\NT NOT 
COLINEAR WITH FIRST TWO POINTS, WITHOUT RO- 
TATION. ALIGN OBJECT WITH ROBOT GRIPPER, 



1 PLACE CURSORS ON SAME TWO FEATURES OF THE OB JECT~| 







MOVE ROBOT INTERACTIVELY AGAIN TO THIRD POINT, 
WITH ROTATION 90 DEGREES, UP TO* 170 DEGREES. 
ALIGN OBJECT WITH GRIPPER. 






PLACE CURSORS ON SAME TWO FEATURES 



SYSTEM COMPUTES AND OISPLAYS CALIBRATION OATA | 



( DONE ) 

Fig. 9 
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Fig. 10 

i 

COMPUTE OBJECT OFFSET FOR ROBOT 



( START ) 







FIND EACH OBJECT FEATURE USING HOUGH TRANS- 
FORM OR OTHER ALGARITHM. 






USE ALL FEATURE FOUND DATA TO COMPUTE OB- 
JECT LOCATION AND ORIENTATION .USING LEAST- 
SQUARES ESTIMATE IF >2 FEATURES. 






COMPUTE OBJEC1 
INAL POSITION BY 
MATRICES- 

REL' 

(HOMOGENEOUS f 
TATION BY THE 1 
LATION BY THE \ 


' POSITION RELA1 
MULTIPLYING HO 

■ FOUND * NOMINA 

MATRIX M REPREi 
\AATRIX_R FOLLOV 
/ECTOR T: 

R II R I2 R I3 T l 
R 2I R 22 R 23 T 2 

R 3I R 32 R 33 T 3 
0 0 0 1 


'IVE TO NOM- 
MOGENEOUS 

* L ~' 

>ENTS A RO- 
VED BY A TRANS- 






TRANSFORM OFFSET FROM CAMERA COORDINATE 
FRAME TO ROBOT COORDINATE FRAME BY MUL- 
TIPLYING HOMOGENEOUS MATRICES' 

ROB = CALIB * REL * CALIB *' 

(CALIB- VISION TO ROBOT CALIBRATION DATA ) 







( DONE ) 
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Fig. II 



HOUGH TRANSFORM 



( START ) 



CLEAR ACCUMULATOR IMAGE TO ALL ZEROES 



SELECT RATABLE ENTRY 



I 



CONVOLVE IMAGE WITHIN WINDOW USING 
FILTER FROM R-TABLE • • . 



TRUNCATE NEGATIVE RESPONSES TO ZERO 



SHIFT CONVOLVED IMAGE BY (X ,Y) VECTOR 
STORED IN R-TABLE • 



DIVIDE SHIFTED EDGE IMAGE 8Y CONSTANT 
RELATED TO NUMBER OF EDGE POINTS 



ADD SHIFTED, SCALED EDGE IMAGE TO ACCUM- 
ULATOR IMAGE 



ypq / ANY 
YES ORE ENTRIES 





\ IN R-TABLE 
7 


IMAGE | 




NO 


PROCESSING] 






LOCATION J 


| LOCATE BRIGHT SPOTS IN ACCUMULATOR IMAGE 







( END ) 
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limited amount of compensation for changes in overall 
GRAY SCALE VISION METHOD AND SYSTEM lighting. 

UTILIZING SAME Most commercial vision systems are based on the 

binary segmentation algorithms originally developed at 
CROSS-REFERENCE TO RELATED 5 SRI International. As the image is scanned, geometric 

APPLICATION measurements are computed for each individually con- 

This invention relates to and hereby incorporates by section region of ones or zeroes. Information is gener- 
reference, United States patent application entitled ated about each region such as area, perimeter, centroid 
"Method and System for Automatically Determining and axes of inertia. This straightforward processing on 
the Position and Attitude of an Object", filed June 4, 10 streams of binary data lends itself to efficient hardware 
1985 and having U.S. Ser. No. 741,123. implementation and, consequently, fairly fast process- 

ing. 

TECHNICAL FIELD Difficulty arises when these systems are utilized in an 

This invention relates in general to machine vision industrial environment Normally, industrial situations 
methods and systems and, in particular, to gray scale 15 are characterized by images that have gray parts on 
machine vision methods and systems utilizing same. gray backgrounds. Finding a threshold which will reli- 
ably separate the object from the background is ex- 
BACKGROUND ART tremely difficult and not always possible. Much creative 

The potential value within the manufacturing indus- development has been done is lighting techniques to 
try for machine vision systems with the flexibility and 20 attempt to extend the binary approach into such appli- 
acuity of human sight is widely recognized. Unfortu- cation areas. This effort is expensive and in many cases, 
nately, the development of a general purpose vision does no t generate systems of the desired industrial reli- 
technology has not been as successful as hoped. No ability. 

single technology has proven to be capable of handling . 
a significant spectrum of applications. Most available 2 * Structured Light 

systems function only within selected market niches and ^ n interesting alternative approach is to project a 
perform disappointingly elsewhere. pattern of light usually a laser line) onto the object in 

The machine vision industry has been involved for question. This bright line of light provides excellent 
the last 10 years in one continuing struggle: How to deal contrast and an easily recognized feature. By projecting 
with the data in the time available. A vision system 30 ^ laser at ^ the displacement of each point on 
works with a digital representation of the scene under ^ , me is made proport i ona i to the depth of the object 
investigation. The scene is digitized into an array of ftt ^ pQint ^ shape of Ae Jme provides additional 
numbers which is roughly 256 data elements square. ^ abQut ±e curvature ^ edges of th e object cut by 
Each 8 bit pixel (picture data element) represents the ^ ^ 

intensity of the reflected light (256 levels) at each point 35 Smictured Hght has feeen effectively applicd m the 
in the scene. This array of 65,536 numbers is i the basic meaS urement. Extremely large systems employ- 

data structure which must be processed by all vision hundreds of projec tor.sensor combinations are used 

sys^mstoobtam mformationfrom the observea^ * automobile bodies. Scanning systems 

The objective for any visio » ^ mounted^f precision tables .have proven effective for 

pixels in the image array in such a manner as to separate *o *" ww x - , . , , , , „ - ti 

the object of interest from the background and the «f°° r measurement of turbme blades and other cnU- 
noise Difficulty arises when a classic : serial I computer is %XXatdy, structured light has its limitations, 
app led to doing operations on such a large block of ^ .V - . * ~\ _ - Antn cn ~. 

date. Most popular 16 to 32 bit microcomputers require The single line of hght which reduces the data so effec- 
10 to 15 seconds to perform a simple noise reducing 45 tively also severely hm* s the information co^t oH^e 
filter operation. This amount of processing time is to- . «nage. tt for example, the ^^^^^ 
tally unacceptable for most industrial tasks. As a result, Presence ^.^^f}^^^^^^^ 
a major objective of those who have sought to deal with of a line additional data must come from additional 
industrial vision problems has been to reduce or sim- images. Efforts to scan the laser line and obtam more 
plify the image data. 50 data result in a significantly increased computational 

Tlie industry to date has been dominated by 3 basic requirement The rotating mirrors necessary for laser 
schemes: binary processing, structured hght and corre- scanning have also not shown themselves to be industn- 
lation. Each method illustrates an approach to reducing ally viable. Therefore, for the general case of processing 
tog data. pictures of stationary objects, structured light has 

55 proven to be of limited usefulness. 

Binary Vision 

, , . . - Correlation 

The most popular image processing method deals 
with binary images. A binary image can be created by A third processing method draws its origin from 
making the assumption that the object and the back- . military work in target tracking. In correlation, a sec- 
ground can be easily separated by the intensity of their 60 tion of the test image is compared with a template of the 
respective reflected light If the distribution of intensi- sought feature (a window or subset of the original un- 
ties is of the proper form (i.e. bimodal), a threshold can age). A difference number is calculated which repre- 
be selected such that all pixels above the threshold sents the qualtity of the match. In operation, a correla- 
(background) are given a 1 and all below (object) a tion system scans a scene where the feature is to be 
zero. Binary imaging accomplishes a data reduction of 65 found by successive application of the template at each 
from 8 bits to one bit per pixel. Some systems do a small point on the image. The surrounding pixels are corn- 
amount of gray level calculation to determine the best pared to the template and a difference number calcu- 
threshold setting for a given picture. This provides a lated. The point on the image where the minimum dif- 
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ference number occurs is presumed to be the location of additional tasks, such as parts inspection which cannot 

the feature sought The matching is inherently rotation- be achieved with conventional robot systems, 

ally sensitive. Some systems add additional rotated tern- Applications of machine vision in robotic systems 

plates to permit some angle offset of the sought feature. have not all been "success" stories. Many plant engi- 

This approach has been very effective in the semicon- 5 neers still face the formidable task of interfacing vision 

ductor industry where complex patterns on the surface systems to robots and devote major engineering effort 

of an integrated circuit form ideal images for correla- to achieve a workable illumination environment, 

tion. Hundreds of systems have been sold for locating Though vision systems have often worked well with 

IC chips prior to automated processing. robots, there is still a broad class of applications which 

Correlation has two limiting characteristics. First the 10 have been marginally operational and which could be 

image to be processed must contain a large number of better handled with different approaches. There are 

contrast transitions to permit the build-up of accurate ^ many potentially successful vision-robot applica- 

statistics about the quality of the match. A piece of sheet ^ ons vet t0 \> e explored. 

metal with one hole resting on another piece of metal Many industrial robotic applications require the accu- 

simply does not have enough features to make a correla- 15 rate pos j t ional location of identical components within a 

tion system effective. Secondly, the only output of the fimte work areJL Typical cxamp!cs migh t be the loading 

system is a single quality of match number much like a or fading of automotive parts from parts trays, the 

"go or no-go" gauge. Finer inspection for continuity or picldng up of apart which has ^ loosdy phoned 

measurement requires the application of different tech- by a partg feeder> Qr the identiflcation i ocation of a 

nologies. 20 wmcn has been cycled into a work cell for a ro- 

Niche Technologies boric* manufacturing operation. Experience has shown 

_ f . , . . . . A , . , that the essential problem in many robotic applications 

The niherent limitations of the technologies de- . ^ , ocation of ^ ^ which . co ^ nei t0 

scribed above illustrate the reason for the rise of niche movement within a *, ane ^ ^ tw(M , imensional 

vision systems. The applications where each technol- 25 ,1,. „~;>; n „ n e X. ^.ij k~j,, 

ogy is effective have multiplied. Applications outside P°* Uon ° f * e w "°* y is completely spec.- 

, , . , , f 1 j u u* u r fied by only three degrees of freedom: The body is only 

the technology niches have been plaqued by high apph- f » VJ • * • «. / 

cation engineering costs and/or limited application. * sl,de . m the tw ° P lanar d J men f ons > to rotate 

Attempts are usually made to generate artificial situa- about a " **? Perpendicular to the plane. 

tions that match the constraints of the technology em- 30 . * °. rd f r ^ computer vision to operate successfully 

ployed. Determining the right technology for a given m thc ""iustnal environment many stringent operating 

job is no small task. It requires an understanding of all of requirements »«t *>c considered. A successful vision 

the available approaches, the product, the surrounding must operate reliably under varying conditions, 

processes and the nature of the specific manufacturing such "changes iniUumination levels and color condi- 

environment in which the system is to operate. Needless 35 tlons of P?* J** svstem should not be dependent 

to say, such an accumulation of skills is rarely found in on complex lighting schemes. In critical applications, 

one organization. Process engineers are faced with the redundancy must be designed m to prevent failures and 

prospect of having a system from one manufacturer at reduce down time. 

the beginning of a line for inspection, another in the vision ^ Processing cannot take a major portion 

middle for gauging and a third at the end for object 40 of cvclc time in robotic applications. Typically, for 

location prior to material handling. Each system ob- a manufacturing operation with a twenty second cycle 

tained from a different manufacturer has a different time, no more than two seconds is possible for vision 

underlying technology and different parts and service processing. 

requirements. The vision system must take a minimum of time and 

45 effort to install, maintain and operate. The time needed 
Industrial Application Needs for Vision Systems to calibrate the vision system itself and to calibrate the 
Vision systems offer immediate advantages in reduc- vision system to a robot must be kept to a minimum. If 
ing tooling and Fixture costs. Precise part location in calibration fixtures are used, they must also be kept 
dunnage is no longer a necessity if part position can be simple, inexpensive, and compact, 
determined by the vision system. Machine vision also 50 Finally, the ultimate aim of the system is still to un- 
makes possible inspection tasks such as dimensional prove manufacturing productivity and provide a cost- 
verification and defect inspection. Vision systems can effective solution to the manufacturing process, 
also provide more increased accuracy and precision Gray Scale Vision 
than many other methods in automation. This added y 

accuracy, coupled with precision robots, can be used to 55 Full gray scale processing ststems hold the promise of 

accomplish tasks which are presently too complex or a more acceptable approach to solving industrial vision 

cost-prohibitive. problems. In 1983 hardware began to emerge that made 

Robots are used in a wide variety of applications in it possible to cost-effectively process gray scale data in 

industry such as painting, spot-welding, and sealing. In times that are acceptable for industrial problem solving, 

the mechanical assembly industry, robots are used for 60 The class of high speed hardware called array proces- 

palletizing/depalletizing parts and unloading parts from sors has long been considered the answer for situations 

trays for assembly operations. In application areas involving large amounts of data, such as in full gray 

where the capabilities of the traditional "blind" robot scale processing. Generally visualized as a matrix of 

fall short, machine vision becomes an integral part of computers, an array processor increases speed by the 

the robot system to assist in end effector guidance. The 65 massively parallel nature of its architecture. Problems 

total-integrated, visually-guided robot system results in can be broken down into components and each compo- 

enhanced manufacturing productivity, improved qual- nent sent to a separate processing node. Growth of 

ity, reduced fixturing costs, and the ability to perform array processors has been slow because of the physical 



06/22/2004, EAST Version: 1.4.1 



4,707,647 

5 6 

size and cost of conventional processing nodes and the and decreasing intensity (gradients) generally found at 

complexity of communications between processors. the edges of objects. Each pixel and its neighborhood 

One set of array processors particularly applicable to pixels are subjected to an arithmetic operation that en- 
image processing operates in the SIMD mode. A SIMD hances edge features of the image, 
mode processor runs a Single Instruction on Multiple 5 The U.S. patent to Rosso! et al No. 4,105,925 dis- 
Data streams. SIMD mode is appropriate for image closes an optical object locator which determines the 
processing since typically the objective is to perform a position and orientation of an object on a conveyor, 
particular operation on an entire image. Unfortunately, Two planes of light intersect at a single transfer slide on 
building such a system with some 65,000 processors, is the conveyor surface. The method assumes a constant 
beyond today's technology. 10 viewing distance and perspective. 

Another hardware architecture which frequently U.S. patents to Blanchard et al U.S. Pat. No. 

operates in SIMD mode makes extensive use of pipe- 4,618,742, Michaud.et al U.S. Pat. No. 3,804,270, Birk et 

lined hardware. In this approach the successive steps al U.S. Pat No. 4,146,924, Pryor et al U.S. Pat. No. 

for processing each element of data are implemented in 4,373,804 and Masaki U.S. Pat. No. 4,380,696 disclose 

a separate piece of hardware. If the process requires 15 machine vision systems which provide visual data 

eight successive operations to complete, then the pipe is which is subsequently processed and input into a form 

constructed of eight successive stages of processing. utilized to alter the preprogrammed path of a peripheral 

The first data element entering the processor completes device, such as a robot, so that the device can perform 

the first stage of processing and enters the second stage, work on an object. 

as the second element is clocked into the first stage. This 20 DISCLOSURE OF THE INVENTION 
process continues until each stage of the pipeline is 

working on a different element of the data stream. The An object of the present invention is to provide an 

first element of data processed emerges from the end of improved method and system to automatically locate an 

the pipe in the normal processing time. The second object in a quick and accurate fashion by identifying 

element of data emerges |th of the time later and so on 25 and locating at least one specific geometric feature of 

until the data stream is exhausted. the object such as an edge, a hole, an ellipse, an arc or 

In 1979-1980 H. T. Kung of Carnegie Mellon Univer- a line, 
sity described a simplified though restricted, form of Another object of the present invention is to provide 
array processor called the systolic array. The systolic an improved method and system for automatically de- 
array design can be used to further adapt pipelines and 30 termining the position and orientation of an object is 
SIMD hardware approaches for fast image processing. rapid fashion from located faetures wherein ambient 
Since a special purpose processor typically receives light conditions are tolerated without significantly de- 
data and returns its results to a more standard host, I/O grading performance. 

processes significantly affect overall performance. Still another object of the present invention is to 
The concept involved in systolic arrays is that of data 35 provide a method and system for automatically generat- 
moving smoothly and rhythmically through the hard- ing information from the position ad orientation of an 
ware. Each data element is accessed only once and object to permit a peripheral device, such as a robot, to 
proceeds through the hardware until all the necessary compensate for any movement of the object from a 
computations required on that element have been per- nominal position and orientation, 
formed. This architecture effectively adapts the con- 40 Yet still another object of the present invention is to 
cepts of array and pipeline processors to the realities of provide a method and system for automatically deter- 
data flow within present day computer systems. mining the position of an object in a factory environ- 
The earliest use of an architecture similar to systolic ment without special lighting, such as structured light- 
arrays came from the activities of the Environmental ing and without the need to use a specially constructed 
Research Institute of Michigan. The process was devel- 45 camera. 

oped to speed the analysis of satellite radar images. In carrying out the above objects and other objects of 
Cytocomputer architecture was developed from this the present invention, a method is provided for auto- 
work and utilizes the concept of nearest neighborhood matically locating an object at a vision station. The 
processing. method includes the steps of generating reference data 
In its early implementations, a 3X3 matrix of pixels 50 relating to at least one feature of an ideal object, and 
was examined to determine the appropriate value for generating a gray-scale digital image containing the 
the center pixel. The data drawn from memory for object to be located at the vision station. The method 
processing resides in a large shift register. During each further includes the step of performing an edge detect- 
cycle, the data moves up one step in the register and a ing algorithm with the reference data and at least a 
processed output value for the new center pixel 55 portion of the digital image to locate the feature within 
emerges from the special hardware. In the Cytocom- the digital image. 

puter, successive processing stages are linked together In one embodiment, the method is utilized to provide 

to permit a sequence of different operations to be exe- offset data for use by a programmed robot controller to 

cuted in turn. This flowing of data through parallel enable the robot controlled by the controller to move to 
processing stages is similar to systolic processing. 60 the object The offset data is the difference between the 

actual and nominal positions of the body at the vision 
Gray Scale Algorithms station. When utilized in this fashion, the method fur- 
Today's systolic processors provide numerous func- ther comprises the additional steps of generating cali- 
tions which are used to analyze the information inside bration data relating the camera to the coordinate frame 
an image such as by utilizing image processing convolu- 65 of the robot, correlating the location of the feature 
tions. These operators allow the application of filters to within the digital image to an offset of the object from 
the image to take away noise and smooth the data. Edge the camera, and transforming the offset of the object 
detecting convolutions deal with patterns of increasing from the camera to the robot frame. 
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In order to completely determine the x and y offsets FIG. 2 is an enlarged, framed view of a scene appear- 

of the object in the image plane, at least two edge points ing on a video monitor of the system illustrating the 

or edge segments with different orientations are re- two-dimensional location of loosely constrained part in 

quired in the reference data. Consequently, preferably, a tray; 

the method utilizes reference data which includes at 5 FIG. 3 is a schematic view of the hardware utilized in 

least two edge points, one on each of a pair of non-par- the subject system; 

allel edge segments of the ideal object. The reference FIG. 4 is a schematic view of the software modules 

data and the digital image are processed together to and tasks utilized with the hardware of FIG. 3 and the 

obtain an accumulator image. The processing step in- flow of data therebetween; 

eludes the step of performing an edge detecting convo- 10 F 1 ®- 5 is a software flowchart of a run time module; 
lution on at least a portion of the digital image for each FIG. * « a software flowchart of a training module; 
edge point. Once the location of at least one localized FIG. 7 is a flowchart illustrating the steps to form a 
bright region in the accumulator image is determined, template for use m the present invention; 
the location of the region corresponding to the location FlG - * is a flowchart iUustrating the steps to set up a 
of one of the features within the digital image is found. 13 w ™° lu *? n mas > for u i, c in «*ject invention; 
To determine orientation, the algorithm must be per- FIG - 9 1S a software flowchart of a calibration mod- 
formed with two different features of the object. u lc j^ _ . . _ , _ _ 

Also, preferably, the reference data includes direc- . *? G - 10 1 ? a soft ^ e fl ™ c * ar ? of a user Usk utlhzed 
tional data for each of the edge points and the process- ^ m *"un time module of FIG. 5; and 
ing step includes the step of shifting the convolved data 20 ?. IG - 11 » a f oftw f f. flowchart of an algorithm corn- 
by an amount and in a direction related to the direc- P nsm S an '^mentation of the Hough transform, 
tional data for each of said edge points to obtain a BEST MODE FOR CARRYING OUT THE 
shifted image. The shifted convolved data is summed to INVENTION 
obtain the accumulator image Finally the location of ^ Referri tQ FIG x there is mmgd a robot 10 
the feature or features within the digital image are cor- which ig ^ tQ move fe an a scaIe 
related with the offset of the object from the camera. machine * ision s ^ 0 * ^ in 

Further m carrymg out the above objects and other accordance wkh ^ nt invent ion. However, it is 

objects of the present invention, a system is provided tQ bfi understood ^ the robot 10 or oth er peripheral 

for carrying out each of the above steps. The system 3Q device is not require d for operation of the present in- 

mcludes a television camera for generating the gray vention 

scale digital image. The vision system as described hereinafter is not only 
Also, the system may be used to automatically pro- « ro bust n (i.e. relatively unaffected by changes in light- 
vide offset data to a peripheral device such as a pro- mgf ob j ect po S i t i on and/or reflectivity of the viewed 
grammed robot controller to enable a robot controlled 35 ob j ect) but also flexible. By being flexible, the system 
by the controller to move to the object even if the ob- ^ re adily adapt to changes of its various components 
ject has moved from its original position and/or orienta- ^ d ^ to ob j ect design variations, 
tion. When utilized with a programmed robot control- tj sc 0 f fte present method and system not only results 
ler, the system further comprises means for transform- m n j gn productivity, but also improved quality. More 
ing the offset of the object from the camera into the 40 sp ecfically, as illustrated in the example of FIG. 1, the 
coordinate frame of the robot. method and system provide accurate positional infor- 
The advantages of the above-described method and ma tion relating to an object located within a Finite work 
system are numerous. For example, such gray scale envelope of the robot 10. In the example shown, the 
vision processing is insensitive to ambient lighting and objects are automotive parts 12 which are to be loaded 
does not require highly specialized lighting. Also, the 45 or unloaded from trays 14. 

processing is insensitive to background and object color other uses of the machine vision system are, for ex- 

and surface texture variations. The gray scale vision ample, picking up of a part which has been loosely 

processing does not require that the object design be positioned by a parts feeder and locating a part which 

modified in order to conform to vision system require- has been cycled into a work cell for a robotic manufac- 

ments. 50 hiring operation. In each of the examples, the method 

The method and system can be used to identify part and system are capable of locating a rigid object which 

styles and accept/reject inspection thresholds can be j s constrained to move within a plane so that the rigid 

selectively checked. Tooling costs can be reduced and object is completely specified by only three degrees of 

retooling programming is relatively easy. Furthermore, freedom (i.e. the object is only free to slide in the two 

industrial standard television cameras can be quickly 55 planar dimensions and to rotate about an axis perpendic- 

replaced and automatically recalibrated in such ma- ular to the plane). 

chine vision method and system. The hardware of a machine vision system con- 
Other advantages of the present invention will be structed in accordance with the present invention is 
readily appreciated as the same becomes better under- collectively indicated at 16 in FIGURE 3. The system 
stood by reference to the following detailed description 60 16 is intended, but not limited to be employed, at one or 
when considered in connection with the accompanying more vision, or work stations, such as the vision station 
drawings. 18 in FIGURE 1. While illustrated mounted on the 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a framed perspective view illustrating the 65 programmable controllers, numerically controlled ma- 
operation and system of the present invention with chinery, other vision systems, plant data management 
respect to the loading and/or unloading of parts from systems and the like. It is also to be understood that the 
part trays; system 16 may provide control functions for several 
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vision or work stations by sharing its time between the image from the camera 20 and places it into a frame 

vision stations. ' buffer preferably measuring approximately 256x240 

The system 16 includes a plurality of cameras 20, one picture elements, each of which consists of an 8-bit 

for each vision station 18. Each camera 20 preferably number representing the brightness of that spot in the 

comprises a conventional solid-state televison camera 5 image. However, it is to be understood that the image 

which provides standard television output signals. may be sampled to any other size, and with an image 

However, it is to be understood that any standard RS- with more or fewer gray levels per pixel. 

170 televison camera may be utilized. A second image buffer is used to store edge-enhanced 

As shown in FIG. 1, the camera 20 is positioned images as they are generated and a third image buffer is 

within a housing 22 mounted on the robot 10. The cam- 10 used as an accumulator buffer. A fourth image buffer is 

era 20 generates an analogue image which contains provided for the shifting of the edge-enhanced images a 

within it an image of the object 12 as illustrated in FIG. predetermined number of pixels up, down, left or right 

2. FIG. 2 is representative of the image generated by the However, it is possible that the shift and add operations 

camera 20 and displayed on a video monitor 24 of the may be done in a single operation to eliminate the neces- 

system 16. It is to be understood that the camera 20 need 15 sity for providing a fourth image buffer, 

not be mounted on the robot 10 but may be placed at a The system 16 also includes a multipoint, functional 

location fixed with respect to the robot 10. array processor which preferably comprises a systolic 

The system 16 does not require any special lighting array processor 32 which, in one mode of operation, 

such as structured lighting. Relatively large variations functions as a two-dimensional hardware convolver to 

in ambient lighting have a minimal effect on the accu- 20 perform a two-dimensional convolution. In another 

racy of the system 16. However, artificial lighting is mode of operation, the array processor 32 performs 

preferable if the camera 20 is located in extreme dark- image addition into the accumulator buffer and in a 

ness such as might be encountered in a poorly illumi- third mode of operation determines the location of the 

nated area. Artificial lighting may also be desirable if pixel having a maximum value in the accumulator 

the vision station 18 regularly experiences large changes 25 buffer. 

in ambient light as might happen with direct sunlight. . The system 16 also includes a single point video rate 

However, in both of the above-noted cases only a few, processor 34 which is selectively coupled to the array 

relatively low-cost light fixtures are necessary for a processor 32 and the image digitizer/image buffers 30 

suitable lighting environment. by image busses 36 and a system bus 38. In one mode of 

When the object 12 viewed by the system 16 is in a 30 operation, the video rate processor 34 performs a histo- 

"0" offset or ideal, nominal position, there is associated gram to determine the value of the maximum pixel in 

with the camera 20, a camera or vision coordinate sys- the accumulator buffer. This value is subsequently 

tern or frame whose origin is located on or near the passed via the system bus 38 to a host computer 40 for 

object 12. The vision coordinate system is assumed to subsequent transmittal to the array processor 32. In 

remain fixed with respect to the viewing camera 20. 35 turn, the array processor 32, as previously mentioned, 

The object 12 also has its own coordinate system which determines the location of the maximum pixel found in 

if fixed with respect to itself. When the object 12 is in its the accumulator buffer. 

"0" offset position, the object coordinate system and the In another mode of operation, the video rate proces- 

vision coordinate system are identical. sor 34 participates in the image digitization process, and 

When the object 12 is moved from its "0" offset posi- 40 also serves as a look-up table to provide a constant for 
tion the rigid body motion that it undergoes it fully reducing the intensity of the edge-enhanced images as 
defined by a transformation relating the vision coordi- will be described in greater detail hereinbelow. , 
nate system to the object coordinate system. This tran- . The system 16 also includes input-output circuits 42 
formation is defined with respect to the vision coordi- to allow the system 16 to communicate with external 
nate system and completely defines the position and 45 peripheral devices such as robots, programmable con- 
orientation of the body coordinate system and, conse- trailers, etc. The system 16 is adapted for expansion of 
quently, the location of the object 12. its input-output functions as schematically illustrated at 

Numerical data describing the offset position of the block 44 and also has capability for memory expansion 

object 12 is illustrated at location 26 on the video moni- as schematically illustrated at block 46. 

tor 24 of FIG. 2. Calibration information relating to the 50 The software of the system 16 is schematically lllus- 

various coordinate systems is generated and thereafter trated in FIG. 4. The system software is preferably 

stored in a mass storage unit 28 of the system 16 as organized in five modules or tasks which perform the 

illustrated in FIG. 9. In particular, the user first places functions of vision, communication, training, calibra- 

an ideal object in view of the camera 20. Then the user tion and runtime control as indicated by blocks 48, 50, 

places cursors on various features such as edge seg- 55 56, 58 and 60, respectively. Lines 54 represent the flow 

ments of the ideal object The robot 10 is then driven of data between the different tasks, modules and exter- 

until it touches the ideal object at which time the robot nal devices. 

position is recorded by the system 16. This same proce- The user task 48 is specifically concerned with the 

dure is repeated until sufficient data is generated to handling of the offsets generated in response to requests 
calculate the various calibration parameters. 60 from the peripheral device. The communication task 50 

A more complete understanding of the interrelation- comprises an input-output interface which provides the 
ship between the various coordinate systems and the other software with the capability to access external 
mathematics which describe the relationship are both devices such as robots, programmable controllers, etc. 
more fully described in the above-noted patent applica- as requested by the peripheral device. A detailed expia- 
tion. 65 nation of the calibration module 58 and the training 

The system 16 includes an image digitizer/image module 56 are illustrated in FIGS. 9 and 6, respectively, 
buffer, schematically indicated at 30 in FIG. 3. The Referring now to FIG. 5 there is illustrated in soft- 
image digitizer portion samples and digitizes the input ware flowchart form, the run time module 60. Briefly, 
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the run time module 60 generates two-dimensional off- 
sets for the object in the field of view of the camera 20. 

Initially, the run time module 60 calls the user task 48, 
as illustrated in block diagram form in FIG. 10 which, 
in turn, calls an edge-detecting algorithm. Preferably, 5 
the algorithm comprises an implementation of the 
Hough transform as illustrated in block diagram form in 
FIG. 11. The Hough transform algorithm looks at the 
object 12 in the field of view of the camera 20 to find the 
image coodinates of certain geometric features of the 10 
object 12 such as holes 13 formed through the object 12. 
Then, the found features are displayed on the video 
monitor 24 as illustrated in FIG. 2. If more feature data 
is generated than is necessary, then a least-squares esti- 
mate routine is utilized to reduce the data. The image IS 
coordinates of the various found features and the previ- 
ously generated calibration data are used to arrive at the 
offset location and orientation of the object 12. 

Thereafter, as further illustrated in FIG; 10, the task 
48 transforms the information into three degrees of 20 
freedom offsets from the previously measured nominal 
position (i.e. found during the training sequences of 
FIG. 6) of the object 12 by multiplying homogenous 
matrices. The task 48 then transforms the vision offsets 
into the frame of reference of a peripheral device, such 25 
as the robot 10, by utilizing the calibration data found 
during the calibration procedure of FIG. 9. Finally, the 
module 60 calls the communication task 50 to send the 
offsets to the controller of the robot 10 which is able to 
use the offsets to compensate its taught path to move to 30 
the object 12. 

Referring now to FIGS. 7, 8 and 11, there is illus- 
trated the Hough algorithm to find each predetermined 
feature of an object within the vision task 48. Briefly, 
the algorithm accepts as its input gray scale digitized 35 
image within which the desired feature or shape, called 
the template, is to be searched for. It produces a trans- 
formed image in which localized bright regions indicate 
the locations of copies of the template shape. The imple- 
mentation of the Hough transform consists of perform- 40 
ing a number of directional, edge-enhancement opera- 
tions on the image; shifting the resulting edge-enhanced 
image by certain amounts in the horizontal and vertical 
directions; and summing the shifted images together in 
the accumulator buffer. 45 

The Hough transform of the present invention as well 
as the generalized Hough transform algorithm is driven 
by a table called an "R-table" which is more fully de- 
scribed at pages 128 through 129 of the book entitled 
"Computer Vision**, authored by Dana H. Ballard and 50 
Christopher M. Brown and published by Prentice Hall 
of Inglewood Cliffs, NJ. in 1982. The steps required to 
set up the R-Table are illustrated in FIG. 7. The first 
step in constructing the R-table is to choose a reference 
point in the template shape for which the Hough trans- 55 
form is to be constructed. The reference point can be 
arbitrary for purposes of the algorithm but when copies 
of the template pattern are found in the digitized image, 
it is the coordinates of this reference point which are 
found. The next step is to choose a number of points 60 
either on the perimeter or some internal edge of the 
template. Preferably, at least six such points are used. 
For each of these edge points, the local gradient direc- 
tion of the image grey-level and the vector, R, to the 
reference point are computed and stored in the R-table. 65 

Referring now to FIG. 11, there is illustrated in flow- 
chart form the implementation of the Hough transform. 
First, all of the points in the two dimensional accumula- 



tor buffer are set to all zero. Second, for each entry in 
the R-table the following operations are performed: (1) 
a direction-sensitive, edge enhancement operation is 
performed on the input image with the direction of 
maximum response set to the direction from the R-table; 
(2) the edge-enhanced image is shifted spatially by the 
amount of the vector R from the R-table, and (3) each 
shifted edge-enhanced image is added to the accumula- 
tor buffer. When this procedure is repeated for all of the 
entries in the R table, the image in the accumulator 
buffer will have its highest value at the points where the 
template shape is located in the original image. 

The edge-enhancement operation preferably utilizes 
the systolic array processor 32 to perform a two-dimen- 
sional convolution on the input image. By way of exam- 
ple, the convolution masks are square arrays of either 
3X3, 5x5 or 7x7 coefficients each of which is prefera- 
bly a 13-bit signed number which is programmable. The 
masks are chosen by sampling the first partial derivative 
in the direction given by theta of the bi-variate Gaussian 
probability distribution with zero mean and equal stan- 
dard deviations along the two coordinate axes. The 
values of the standard deviation can be chosen over a 
wide range, but best results have been obtained with 
values of 0.85, 1.05 and 1.50 for the3x3, 5x5and7x7 
convolution masks, respectively. 

Because the first directional derivative of the bi-vari- 
ate Gaussian probability densing function is symmetric, 
the sum of the convolution mask coefficients is zero. 
The coefficients are all multiplied by a constant factor 
in order to make the sum of their squares equal to some 
given gain factor. This gain factor depends on the con- 
trast of the input image and is automatically computed 
by software when the R-table is set up. 

The convolution masks are preferably precomputed 
as illustrated in FIG. 8 for each gradient direction 
stored in the R-table since computing the convolution 
masks is a relatively slow operation involving floating 
point arithmetic. Consequently, when the algorithm is 
executed, the masks merely have to be retrieved from 
the R-table and passed to the hardware convolver (i.e. 
systolic array processor 32). 

Response of image edges to convolutions with the 
first derivative of the Gaussian filter is strongest for 
edges aligned in the direction of te filter and falls off 
with the consine of the angle of misalignment. Edges 
that are plus or minus 90° from the desired direction 
thus give zero response and edges of opposite polarity 
give negative responses. As illustrated in FIG. 11, nega- 
tive responses are set to zero. The use of such a direc- 
tion-sensitive edge detector has a distinct advantage 
over the use of an isotropic one, such as the Lapacian, 
or gradient magnitude edge detectors in that far fewer 
undesired edges are generated. Such undesired edges 
may accidentally reinforce other edges to produce spu- 
rious peaks in the accumulator image. 

Because there are only eight bits available for each 
number in the accumulator buffer, care must be taken 
not to exceed the maximum available when adding the 
shifted, edge-enhanced images together. On the other 
hand, the peaks in the accumulator buffer are preferably 
as bright as possible. Consequently, as illustrated in 
FIG. 8, the gain factors of the edge detecting filters are 
adjusted by software to give a response close to, but not 
exceeding, the maximum pixel value for the point in the 
image which gives the highest response. Second, as 
illustrated in FIG. 11, pixel intensities of the edge- 
enhanced image are divided by a constant by using an 
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8-bit-in, 8-bit-out lookup tabic residing in the video rate 
processor 34 before they are added to the image in the 
accumulator buffer. The constant used is the number of 
R-table points which are expected to coincide with a 
copy of the template pattern present in the image. 5 

Once all of the shifted images have been added to- 
gether in the accumulator buffer, the video rate proces- 
sor 34 performs a histogram on the data therein to deter- 
mine the value of the maximum pixel. This value is 
subsequently utilized by the systolic array processor 32 10 
to determine the location of the maximum picture ele- 
ment which is related to the coordinates of the previ- 
ously chosen reference point. 

In identifying the location of circles and circular arcs, 
the R-table points are preferably equally spaced about 15 
the perimeter of the circle. Also, the edge-detector 
directions are all tangent to the circle and perpendicular 
to the shift vectors which are all radii of a circle having 
the reference point as its center. Because of this, the 
algorithm is insensitive to rotation of the input image, 20 
and, consequently; a part may be recognized and ori- 
ented merely by finding two or three holes or arcs with 
the algorithm. 

The above-noted system and method is capable of 
examining and determining the position and attitude of 25 
an object at a vision station using a single camera with- 
out the need for special lighting. Object information can 
thereafter be communicated to any type of peripheral 
device, such as a robot The gray scale machine vision 
system of the present invention does not require use of 30 
highly specialized or complex hardware and is robust 
(i.e. tolerates small variations in work pieces, camera 
adjustment and ambient light without significant degra- 
dation of system performance.) The system tolerates 
noisy, low-contrast images and is easily calibrated. 35 

The invention has been described in an illustrative 
manner, and, it is to be understood that, the terminology 
wheh has been used is intended to be in the nature of 
words of description rather than of limitation. 

Obviously, many modifications and variations of the 40 
present invention are possible in light of the above 
teachings. 

What is claimed is: 

1. Method for automatically locating an object at a 
vision station, said method comprising the steps of: 45 

generating reference data relating to at least two 
features of an ideal object, the data including at 
least two edge points, one of said points being on 
each of a pair of non-parallel edge segments 
wherein the reference data includes directional 50 
data for each of the edge points; 

generating a gray-scale digital image containing the 
objects to be located at the vision station; 

processing the reference data and the ditital image 
together to obtain an accumulator image, said pro- 55 
cessing step including the steps of performing an 
edge-detecting convolution on at least a portion of 
the digital image for each edge point and shifting 
the convoluted data by an amount and in a direc- 
tion related to the directional data for each of said 60 
edge points to obtain a shifted image; and 

detennining the location of at least one localized 
bright. region in the accumulator image, the loca- 
tion of said region corresponding to the location of 
one of the features within the digital image. 65 

2. Method of automatically generating offset data for 
use by a programmed robot controlled to enable a robot 
controlled by the controller to move to an object at the 
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vision station including a camera, the offset data relat- 
ing the position of the object to the coordinate frame of 
the robot, the method comprising the step of: 

generating calibration data relating the camera to the 
coordinate frame of the robot; 

generating reference data relating to at least two 
features of an ideal object, the data including at 
least two edge points one of said points being on 
each of a pair of non-parallel edge segments 
wherein the reference data includes directional 
data for each of the edge points; 

generating a gray-scale digital image containing the 
object to be located at the vision station; 

processing the reference data and the digital image 
together to obtain an accumulator image, said pro- 
cessing step including the steps of performing an 
edge-detecting convolution on at least a portion of 
the digital image for each edge point and shifting 
the convoluted data by an amount and in a direc- 
tion related to the directional data for each of said 
edge points to obtain a shifted image; 

determining the location of at least one localized 
bright region in the accumulator image, the loca- 
tion of said region corresponding to the location of 
one of the features within the digital image; 

correlating the location of the features within the 
digital image of an offset of the object from the 
camera; and 

transforming the offset of the object from the camera 
to the robot frame. 

3. The method as claimed in claim 1 or claim 2 
wherein said processing step includes the step of sum- 
ming the shifted convoluted data to obtain the accumu- 
lator image. 

4. The method as claimed in claim 3 including the 
step of scaling the gray levels of the shifted image by an 
amount related to the number of edge points prior to 
said step of summing. 

5. The method as claimed in claim 1 wherein the 
digital image is generated by a camera and further com- 
prising the step of correlating the location of the fea- 
tures within the digital image to an offset of the object 
from the camera. 

6. The method as claimed in claim 5 further compris- 
ing the steps of generating calibration data relating the 
coordinate frame of the camera to the coordinate frame 
of a peripheral device located at the vision station and 
transforming the offset of the object from the camera to 
the device frame. 

7. The method as claimed in claim 1 or claim 2 
wherein said edge segments are defined by entries in an 
R-table having a predetermined reference point and the 
edge points and wherein for each edge point entry there 
is defined a local gradient direction and a vector to the 
reference point 

8. A gray-scale vision system for automatically locat- 
ing an object at a vision station comprising: 

means for storing reference data relating to at least 
two features of an ideal object, the data including 
at least two edge points one of said edge points 
being on each of a pair of non-parallel edge seg- 
ments wherein the reference data includes direc- 
tional data for each of the edge points; 

means for generating a gray-scale digital image con- 
taining the object to be located at the vision station, 
said means for generating including a television 
camera; 
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means for processing the reference data and the digi- 
tal image together to obtain an accumulator image, 
said processing means including means for per- 
forming an edge-detecting convolution on at least a 
portion of the digital image for each edge point and 
shifting the convoluted data by an amount and in a 
direction related to the directional data for each of 
said edge points to obtain a shifted image; and 

means for determining the location of at least one io 
localized bright region in the accumulator image, 
the location of said region corresponding to the 
location of one of the features within the digital 
image. 

9. The system as claimed in claim 8 further compris- 
ing means for correlating the location of the features 
within the digital image to an offset of the object from 
the camera. 

10. A system for automatically generating offset data 20 
for use by a programmed robot controller to enable a 
robot controlled by the controller to move to an object 

at a vision station, the offset data relating the position of 
the object to the coordinate frame of the robot, the 
system comprising: ^5 
means for storing reference data relating to at least 
two features of an ideal object, the data including 
at least two edge points, one of said edge points 
being on each of a pair of non-parallel edge seg- 30 
ments wherein the reference data includes direc- 
tional data for each of the edge points; 
means for generating a gray-scale digital image con- 
taining the object to be located at the vision station, 

35 



said means for generating including a television 
camera; 

means for storing calibration data relating the camera 
to the coordinate frame of the robot; 

means for processing the reference data and the digi- 
tal image together to obtain an accumulator image, 
said processing means including means for per- 
forming an edge detecting convolution on at least a 
portion of the digital image for each edge point and 
shifting the convoluted data by an amount and in a 
direction related to the directional data for each of 
said edge points to obtain a shifted image; 

means for determining the location of at least one 
localized bright region in the accumulator image, 
the location of said region corresponding to the 
location of one of the features within the digital 
image; 

means for correlating the location of the features 
within the digital image to an offset of the object 
from the camera; and 

means for transforming the offset of the object from 
the camera to the robot frame. 

11. The system as claimed in claims 8, 9 or 10 wherein 
said means for processing the reference data and the 
digital data includes a gray scale processor for perform- 
ing each of the convolutions. 

12. The system as claimed in claim 11 wherein at least 
one of said convolution is direction-sensitive and com- 
prises a two-dimensional convolution of the digital im- 
age. 

13. The system as claimed in claim 12 wherein said 
gray scale processor comprises a systolic array proces- 
sor. 
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